Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dmsghttp logserver route #1579

Merged
merged 36 commits into from
May 25, 2023
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
0219ce8
add explicit dmsgpty whitelist
0pcom May 13, 2023
30d4055
add WIP env detection to skywire-cli config gen
0pcom May 14, 2023
2136e61
stashing changes
0pcom May 15, 2023
28cf999
add whitelist for survey collecton over dmsghttp
0pcom May 15, 2023
7f5f6a7
Merge branch 'dmsghttp-wl' into config-envs
0pcom May 15, 2023
13d67d6
small fixes and formatting optimizations
0pcom May 15, 2023
cc0f8b0
make format
0pcom May 15, 2023
598a398
add nolint
0pcom May 15, 2023
f4a5b85
remove unused variable, use print-style function instead of printf-style
0pcom May 15, 2023
664b652
merge develop
0pcom May 19, 2023
1c6cd55
update config gen with more flags ; support survey whitelist, route &…
0pcom May 21, 2023
68adf7b
update vendor
0pcom May 21, 2023
22a5d5c
update vendor
0pcom May 21, 2023
8eb076e
update vendor dependencies and go.mod
0pcom May 22, 2023
9728b5b
add error checks, comments
0pcom May 22, 2023
bb13cd4
fix merge conflicts
0pcom May 22, 2023
26ee1f8
update vendor
0pcom May 22, 2023
7662a3d
fix vendor deps & update go.mod
0pcom May 22, 2023
e19f5de
make format & fix test error
0pcom May 22, 2023
4361174
remove viper from config gen
0pcom May 23, 2023
9a3e04e
make format
0pcom May 23, 2023
289b167
prepare to revert some changes
0pcom May 23, 2023
93a7f7e
prepare to revert some changes
0pcom May 23, 2023
f9df75d
fix panic on visor startup
0pcom May 23, 2023
e94d07b
fix ci errors
0pcom May 23, 2023
3287ca4
fix ci errors
0pcom May 23, 2023
15fc1c6
fix golangci-lint errors for version 1.52.1
0pcom May 23, 2023
7a7cdab
update golangci-lint version used by github-actions CI in test workfl…
0pcom May 23, 2023
5174f48
fix CI darwin error
0pcom May 23, 2023
49ce0d9
update visor init and visorconfig values with SurveyName
0pcom May 24, 2023
041a06a
merge develop
0pcom May 24, 2023
a5e5dc4
use config gen from develop
0pcom May 24, 2023
9656094
fix comment
0pcom May 24, 2023
0612c0d
fix spelling on help menu
0pcom May 25, 2023
8a18313
changes per review ; add reward.txt file authenticated endpoint for f…
0pcom May 25, 2023
7322064
make format check
0pcom May 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion pkg/skyenv/skyenv.go
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,8 @@ const (
RewardFile string = "reward.txt"

// NodeInfo is the name of the survey file
NodeInfo string = "node-info.json"
NodeInfo string = "node-info.json"
SurveyName string = "node-info" // SurveyName ...
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to define the HTTP path here. We dont need to change it frequently and can just hardcode it as its used exactly once.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will change

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've dropped the extra declaration SurveyName, but instead of directly hardcoding I've opted for this implementation:

	//Derive the surveyName from visorconfig.NodeInfo by dropping the ".json" suffix
	surveyName := strings.TrimSuffix(string(visorconfig.NodeInfo), ".json")
	// This survey endpoint generates the survey as a response
	authRoute.GET("/"+surveyName, func(c *gin.Context) {

)

// SkywireConfig returns the full path to the package config
Expand Down
39 changes: 4 additions & 35 deletions pkg/visor/cmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,6 @@ var (
// visorBuildInfo holds information about the build
visorBuildInfo *buildinfo.Info
dmsgServer string
rawSurvey bool
isStoreLog bool
isForceColor bool
)
Expand Down Expand Up @@ -117,7 +116,6 @@ func init() {
hiddenflags = append(hiddenflags, "hv")
RootCmd.Flags().BoolVarP(&disableHypervisorPKs, "xhv", "k", false, "disable remote hypervisors \u001b[0m*")
hiddenflags = append(hiddenflags, "xhv")
// initAutoPeerFlags()
RootCmd.Flags().StringVarP(&logLvl, "loglvl", "s", "", "[ debug | warn | error | fatal | panic | trace ] \u001b[0m*")
hiddenflags = append(hiddenflags, "loglvl")
RootCmd.Flags().StringVarP(&pprofMode, "pprofmode", "q", "", "[ cpu | mem | mutex | block | trace | http ]")
Expand All @@ -130,12 +128,10 @@ func init() {
hiddenflags = append(hiddenflags, "syslog")
RootCmd.Flags().StringVarP(&completion, "completion", "z", "", "[ bash | zsh | fish | powershell ]")
hiddenflags = append(hiddenflags, "completion")
RootCmd.Flags().BoolVar(&rawSurvey, "raw-survey", false, "survey will generate and store decrypted if pass this flag")
hiddenflags = append(hiddenflags, "raw-survey")
RootCmd.Flags().BoolVarP(&isStoreLog, "store-log", "l", false, "store all logs to file")
hiddenflags = append(hiddenflags, "store-log")
RootCmd.Flags().BoolVar(&isForceColor, "force-color", false, "set force coler true in loggers")
hiddenflags = append(hiddenflags, "force-color")
RootCmd.Flags().BoolVarP(&isStoreLog, "storelog", "l", false, "store all logs to file")
hiddenflags = append(hiddenflags, "storelog")
RootCmd.Flags().BoolVar(&isForceColor, "forcecolor", false, "force color logging when out is not STDOUT")
hiddenflags = append(hiddenflags, "forcecolor")
RootCmd.Flags().BoolVar(&all, "all", false, "show all flags")
for _, j := range hiddenflags {
RootCmd.Flags().MarkHidden(j) //nolint
Expand All @@ -144,33 +140,6 @@ func init() {

}

//omit due to possible panic
/*
func initAutoPeerFlags() {
localIPs, err := netutil.DefaultNetworkInterfaceIPs()
if err != nil {
logger.WithError(err).Warn("Could not determine network interface IP address")
if len(localIPs) == 0 {
localIPs = append(localIPs, net.ParseIP("192.168.0.1"))
}
}
RootCmd.Flags().StringVarP(&autoPeerIP, "hvip", "l", trimStringFromDot(localIPs[0].String())+".2:7998", "set hypervisor by ip")
hiddenflags = append(hiddenflags, "hvip")
isDefaultAutopeer := false
if os.Getenv("AUTOPEER") == "1" {
isDefaultAutopeer = true
}
RootCmd.Flags().BoolVarP(&isAutoPeer, "autopeer", "m", isDefaultAutopeer, "enable autopeering to remote hypervisor")
hiddenflags = append(hiddenflags, "autopeer")
}
func trimStringFromDot(s string) string {
if idx := strings.LastIndex(s, "."); idx != -1 {
return s[:idx]
}
return s
}
*/

// RootCmd contains the help command & invocation flags
var RootCmd = &cobra.Command{
Use: "visor",
Expand Down
2 changes: 1 addition & 1 deletion pkg/visor/init.go
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,7 @@ func initDmsgHTTPLogServer(ctx context.Context, v *Visor, log *logging.Logger) e
}
}

lsAPI := logserver.New(logger, v.conf.Transport.LogStore.Location, v.conf.LocalPath, v.conf.DmsgHTTPServerPath, whitelistedPKs, printLog)
lsAPI := logserver.New(logger, v.conf.Transport.LogStore.Location, v.conf.LocalPath, v.conf.DmsgHTTPServerPath, whitelistedPKs, v.conf.PK, printLog)

lis, err := dmsgC.Listen(visorconfig.DmsgHTTPPort)
if err != nil {
Expand Down
47 changes: 43 additions & 4 deletions pkg/visor/logserver/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,17 @@ import (
"net/http"
"os"
"path/filepath"
"strings"
"time"

"github.com/gin-gonic/gin"
coincipher "github.com/skycoin/skycoin/src/cipher"

"github.com/skycoin/skywire-utilities/pkg/buildinfo"
"github.com/skycoin/skywire-utilities/pkg/cipher"
"github.com/skycoin/skywire-utilities/pkg/httputil"
"github.com/skycoin/skywire-utilities/pkg/logging"
"github.com/skycoin/skywire/pkg/visor/visorconfig"
)

// API register all the API endpoints.
Expand All @@ -28,7 +31,7 @@ type API struct {
}

// New creates a new API.
func New(log *logging.Logger, tpLogPath, localPath, customPath string, whitelistedPKs []cipher.PubKey, printLog bool) *API {
func New(log *logging.Logger, tpLogPath, localPath, customPath string, whitelistedPKs []cipher.PubKey, visorPK cipher.PubKey, printLog bool) *API {
api := &API{
logger: log,
startedAt: time.Now(),
Expand All @@ -50,16 +53,52 @@ func New(log *logging.Logger, tpLogPath, localPath, customPath string, whitelist
if len(whitelistedPKs) > 0 {
authRoute.Use(whitelistAuth(whitelistedPKs))
}
// note that the survey only exists / is generated if the reward address is set
authRoute.StaticFile("/node-info.json", filepath.Join(localPath, "node-info.json"))
// note that the survey FILE only exists / is generated if the reward address is set
authRoute.StaticFile("/"+visorconfig.NodeInfo, filepath.Join(localPath, visorconfig.NodeInfo))
// This survey endpoint generates the survey as a response
authRoute.GET("/"+visorconfig.SurveyName, func(c *gin.Context) {
var rewardAddress string
var cAddr coincipher.Address
//check for reward address
rewardAddressBytes, err := os.ReadFile(localPath + "/" + visorconfig.RewardFile) //nolint
if err == nil {
//remove any newline from rewardAddress string
rewardAddress = strings.TrimSuffix(string(rewardAddressBytes), "\n")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just use strings.TrimSpace please.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed on next commit

//validate the skycoin address
cAddr, err = coincipher.DecodeBase58Address(rewardAddress)
if err != nil {
log.WithError(err).Error("Invalid skycoin reward address " + rewardAddress)
}
log.Debug("Skycoin reward address: ", cAddr.String())
}
survey, err := visorconfig.SystemSurvey()
if err != nil {
log.WithError(err).Error("Could not read system info.")
}
survey.PubKey = visorPK
survey.SkycoinAddress = cAddr.String()
// Print results.
s, err := json.MarshalIndent(survey, "", "\t")
if err != nil {
log.WithError(err).Error("Could not marshal system survey to json.")
}

c.Header("Content-Type", "application/json")
c.Writer.WriteHeader(http.StatusOK)

_, err = c.Writer.Write([]byte(s))
if err != nil {
httputil.GetLogger(c.Request).WithError(err).Errorf("failed to write json response")
}
Comment on lines +85 to +97
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I mentioned this before, but you can literally just do:

c.JSON(http.StatusOK, gin.H{survey})

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am intentionally using generic c.Writer.Write methods instead of gin's type-specific methods.

It makes it more interchangeable with standard library http server in case we ever decide to move away from gin.

})

r.GET("/health", func(c *gin.Context) {
api.health(c)
})

// serve transport log files ; then any files in the custom path
r.GET("/:file", func(c *gin.Context) {
// files with .csv extension are likely transport log files
// files with .csv extension are **likely** transport log files
if filepath.Ext(c.Param("file")) == ".csv" {
// check transport logs dir for the file, and serve it if it exists
_, err := os.Stat(filepath.Join(tpLogPath, c.Param("file")))
Expand Down
81 changes: 0 additions & 81 deletions pkg/visor/visorconfig/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,84 +54,3 @@ func (d *Duration) UnmarshalJSON(b []byte) error {
return errors.New("invalid duration")
}
}

// VisorConfig is every field of the config
type VisorConfig struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the old visorconfig, right? Any reason we are deleting it? Because its not used anymore and we dont expect users to have to upgrade from that?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that was something that I added there a long time ago that i think was used briefly but then not used

Version string `json:"version"`
Sk string `json:"sk"`
Pk string `json:"pk"`
Dmsg struct {
Discovery string `json:"discovery"`
SessionsCount int `json:"sessions_count"`
Servers []struct {
Version string `json:"version"`
Sequence int `json:"sequence"`
Timestamp int `json:"timestamp"`
Static string `json:"static"`
Server struct {
Address string `json:"address"`
AvailableSessions int `json:"availableSessions"`
} `json:"server"`
} `json:"servers"`
} `json:"dmsg"`
Dmsgpty struct {
DmsgPort int `json:"dmsg_port"`
CliNetwork string `json:"cli_network"`
CliAddress string `json:"cli_address"`
} `json:"dmsgpty"`
SkywireTCP struct {
PkTable interface{} `json:"pk_table"`
ListeningAddress string `json:"listening_address"`
} `json:"skywire-tcp"`
Transport struct {
Discovery string `json:"discovery"`
AddressResolver string `json:"address_resolver"`
PublicAutoconnect bool `json:"public_autoconnect"`
TransportSetupNodes interface{} `json:"transport_setup_nodes"`
} `json:"transport"`
Routing struct {
SetupNodes []string `json:"setup_nodes"`
RouteFinder string `json:"route_finder"`
RouteFinderTimeout string `json:"route_finder_timeout"`
MinHops int `json:"min_hops"`
} `json:"routing"`
UptimeTracker struct {
Addr string `json:"addr"`
} `json:"uptime_tracker"`
Launcher struct {
ServiceDiscovery string `json:"service_discovery"`
Apps []struct {
Name string `json:"name"`
AutoStart bool `json:"auto_start"`
Port int `json:"port"`
Args []string `json:"args,omitempty"`
} `json:"apps"`
ServerAddr string `json:"server_addr"`
BinPath string `json:"bin_path"`
} `json:"launcher"`
Hypervisors []interface{} `json:"hypervisors"`
CliAddr string `json:"cli_addr"`
LogLevel string `json:"log_level"`
LocalPath string `json:"local_path"`
StunServers []string `json:"stun_servers"`
ShutdownTimeout string `json:"shutdown_timeout"`
RestartCheckDelay string `json:"restart_check_delay"`
IsPublic bool `json:"is_public"`
PersistentTransports interface{} `json:"persistent_transports"`
Hypervisor struct {
DbPath string `json:"db_path"`
EnableAuth bool `json:"enable_auth"`
Cookies struct {
HashKey string `json:"hash_key"`
BlockKey string `json:"block_key"`
ExpiresDuration int64 `json:"expires_duration"`
Path string `json:"path"`
Domain string `json:"domain"`
} `json:"cookies"`
DmsgPort int `json:"dmsg_port"`
HTTPAddr string `json:"http_addr"`
EnableTLS bool `json:"enable_tls"`
TLSCertFile string `json:"tls_cert_file"`
TLSKeyFile string `json:"tls_key_file"`
} `json:"hypervisor"`
}
2 changes: 1 addition & 1 deletion pkg/visor/visorconfig/v1.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ type Transport struct {
Discovery string `json:"discovery"`
AddressResolver string `json:"address_resolver"`
PublicAutoconnect bool `json:"public_autoconnect"`
TransportSetupPKs []cipher.PubKey `json:"transport_setup_nodes"`
TransportSetupPKs []cipher.PubKey `json:"transport_setup"`
LogStore *LogStore `json:"log_store"`
StcprPort int `json:"stcpr_port"`
SudphPort int `json:"sudph_port"`
Expand Down
3 changes: 3 additions & 0 deletions pkg/visor/visorconfig/values.go
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,9 @@ var (
// NodeInfo is the name of the survey file
NodeInfo = skyenv.NodeInfo

// SurveyName is the name of the system survey (file) without the extension
SurveyName = skyenv.SurveyName

// RewardFile is the name of the file containing skycoin reward address
RewardFile = skyenv.RewardFile
)
Expand Down